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Abstract 

Research frequently reports student dissatisfaction with feedback in higher education. Large class sizes and 
modularization challenge teachers in providing useful feedback. Most of these studies have investigated 
student perceptions of written feedback in coursework, and few attempts have been made considering 
feedback in face-to-face contexts such as bachelor's degree projects. This study aims to enrich our 
understanding of students' perception of feedback in the context of supervision of bachelor's degree 
projects using Karlsen's (2015) PLUS model to systematise factors that help improve their utilisation of 
feedback in learning. Qualitative interviews were used to collect data from two bachelor student cohorts 
doing their projects as part of industrial design programmes and computer science at a mid-sized Norwegian 
university college (n=28). Results indicate that students generally find teachers' feedback more useful than 
useless. In addition to the students own attitudes towards assessment, they report that how they perceive 
the supervisors' trustworthiness matters when utilising feedback. 
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Introduction 

Feedback is the basis of formative assessment (Black, 2008; Sadler, 1989) and is central to higher educational 
processes (Ramsden, 2003). It is a complex phenomenon designed for various pragmatic and epistemological 
purposes, depending on the learning environment (Poulos & Mahony, 2008). Recently, in higher education, a 
shift is taking place from a cognitivist view of feedback to a so-called co-constructivist and sustainable per¬ 
spective, emphasising the dynamic and iterative nature of learning (Evans, 2013; Boud & Soler, 2016; Ajjawi 
& Boud, 2017). In the co-constructive perspective, feedback is an integral part of learning and is defined as 
’an active, constructive process whereby learners set goals for their learning and then attempt to monitor, 
regulate, and control their cognition’ (Pintrich & Zusho, 2002, p. 250, cited in Carless, Salter, Yang & Lam, 
2011, p. 396). This study examines how design students at a mid-sized Norwegian university college reflect 
on the utility of teachers' feedback in the context of capstone project supervision in design education. 

Improving feedback is a central goal of the Norwegian Quality Reform 1 initiated at the beginning of the 2003 
academic year to align Norwegian higher education to the Bologna process. The reform brought many 
changes to educational institutions in Norway, including a reinforced focus on formative assessment. After 
the reform, students have consistently confirmed that they appreciate being followed up closely, but give 


1 White paper/ Official government-issued report, 27 (2000-2001) 
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low scores when assessing the quality of the feedback they receive (Aamodt, Hovdhaugen, & Opheim, 2006; 
Bakken, Damen, & Keller, 2015; Damen, Keller, Hamberg, & Bakken, 2016). International student surveys, 
such as the Australian Course Experience Questionnaire (James, Krause, & Jennings, 2010) and the British 
National Student Report (HEFCE, 2010), show similar results. Students' dissatisfaction with feedback could 
be due to many factors: teachers may have less time to give individualized feedback because of growing 
class sizes (Hounsell, 2003); There may be conflicts between the many roles that an academic teacher must 
manage (Tuck, 2012); The modularisation of study programmes may impend longer feedback cycles 
(Hounsell, 2003; Robson, Leat, Wall, & Lofthouse, 2013). Other factors reported in the literature of why stu¬ 
dents find the feedback they receive wanting, is that the students themselves are not capable of making 
good use of feedback (Elwood & Bode, 2014; Varlander, 2008; Winstone, Nash, Rowntree, & Parker, 2016). 
There is little reason for teachers to invest in giving feedback if students do not consider the feedback worth¬ 
while. The students' positive attitude towards their feedback is necessary for developing sustainable feed¬ 
back practices, with self-regulation at its core (Carless, Salter, Yang & Lam, 2011, p. 396). Nicol (2010) sup¬ 
ports this claim by writing ’While the quality of the comments is important, the quality of the students' inter¬ 
action with those comments is equally, and perhaps more, important' (p. 503). Carless (2006), in his compre¬ 
hensive analysis of written feedback on assignments in a Hong Kong context, highlights another claim that 
that teachers often believe they give more useful feedback than students feel they receive (p. 225). 

New perspectives on feedback practice in addition to structural reforms in higher education together give 
ample reasons for conducting more research on feedback (Carless et al., 2011; Evans, 2013). Among the 
many areas of research on feedback, McGrath, Taylor, and Pychyl (2011) point out, it is '...of the utmost im¬ 
portance to determine the type of feedback students find most beneficial' (p. 1), and students are in a good 
position to determine if the feedback is beneficial as they are the recipients of the feedback (Price, Handley, 
Millar & O'Donovan, 2010). However, students may not always remember or able to articulate the type of 
feedback they have received. Although a growing body of research explores students' perceptions of written 
feedback in coursework, few studies consider oral, face-to-face feedback in the context of supervised cap¬ 
stone projects. The objective of this study is to critically examine how students perceive the feedback re¬ 
ceived in their projects. The aim of this study is to enrich our knowledge of how bachelor students utilize 
their oral feedback. This study was designed to answer the following research questions: 

1. What types of oral feedback comments from supervisors do bachelor students find most useful when 
completing their final capstone projects? 

2. What conditions must be met, according to the students, for them to benefit from this feedback? 

In the following, I will present the analytical framework and research design used to investigate these ques¬ 
tions, followed by the results, a discussion of these results, and a conclusion which I believe contribute to 
identifying and systematising factors that will improve the students' feedback utilisation in their learning 
processes. 

Analytical framework and Research Design 

To be able to answer the research questions, open-ended in-depth interview data were collected and then 
analysed using the PLUS model (Karlsen, 2015). This model is multidimensional, flexible and distinguishes 
four aspects of feedback—form, focus, purpose and temporality. Each aspect is defined by 2-8 categories or 
codes (see Figure 1). The connections between the aspects are decided by how the aspects and codes are 
combined. The combination form a 'pattern of feedback', expressed as 'form/focus/purpose/temporality'. 
The model allows for 192 different combinations (8 x 4 x 3 x 2). The following summarises the PLUS model. 
However, for a full description, see Karlsen (2015). 
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The first aspect, the form of the feedback, captures different ways teachers formulate and deliver feedback 
to students. The model describes eight forms of feedback: correction, explanation, judgement, suggestion, 
emotionally charged response, brainstorm, question and interpretation. The second aspect, the focus of the 
feedback, concerns what the feedback addresses, for example, a method or strategy students employ, the 
argumentation of an essay or the students' capacity for self-regulation. Similar to de Kleijn, Mainhard, Mei- 
jer, Brekelmans, and Pilots (2013) and Jolly and Boud (2013), the PLUS model uses Hattie and Timperley's 
(2007, p. 86) four levels at which feedback operates to define the focus of the feedback: product, process, 
self-regulation and person. The third aspect, the purpose of the feedback, categorises what teachers want 
students to learn, for example, to help the students see the beauty in a painting, to be able to follow a code 
of professional conduct or to become more motivated. The model uses established classifications of educa¬ 
tional objectives to define the purpose of feedback as consisting of the cognitive, affective and psychomotor 
domains (Bloom, 1956; Krathwohl, 2002; Krathwohl, Bloom, & Masia, 1964; Simpson, 1966). The fourth as¬ 
pect, the temporality of the feedback, describes the temporal direction of feedback vis-a-vis its focus. Feed¬ 
back is commonly distinguished as either 'feed-back' or 'feed-forward' (see, e.g. Nicol, 2010; Sadler, 2010; 
Walker, 2013). These are the terms used for the temporality of feedback in the PLUS model. 


A Type of Feedback 


FORM 


FOCUS 

k_> 

/" 


PURPOSE TEMPORALITY 



Figure 1: The feedback classification scheme used in coding 

As mentioned earlier, the PULS model was used analysing the data gathered in this research. The data com¬ 
prises of face-to-face interviews of 28 bachelor's students; twenty taking the bachelor's degree project 
course as part of an industrial design programme, and eight taking a design course as part of a computer sci¬ 
ence programme. In both bachelor programs during their sixth semester, the students must complete a cap¬ 
stone project that ends with an exhibition of the projects, an EXPO. The aim of the courses is to develop the 
students' competence and self-esteem in carrying out real projects. Throughout the course, they practice 
working in teams systematically, according to recognized methods and models for project management, 
problem solving, development and evaluation. During these courses, students from both programs work in 
groups of 3-4 to design and make prototypes to solve genuine work-based challenges as defined in coopera¬ 
tion with the academic supervisor and an external company (see last column table 1). The 28 students 
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worked nine different bachelor projects summarised in table 1. The first column gives an overview of the 
end products each of the groups designed. The second column divides the projects in two cohorts based on 
whether the cohort is an industrial design (group 1-7) or an interaction design (group 8-9). 


TABLE 1: CAPSTONE PROJECTS 


Gr. 

END PRODUCT 

TYPE OF PROJECT 

EXTERNAL COMPANY 

1 

A prototype of a veterinary examination ta¬ 
ble 

Industrial design 

Norwegian University of Life Sci- 

ences 

2 

A prototype for mobile navigation- and 
communication device for infantry soldiers 

Industrial design 

Norwegian Defence Research Es¬ 
tablishment (FFI) 

3 

A car entering the Shell Eco Marathon race 

Industrial design 

Shell Eco Marathon / Shell Global 

4 

A prototype of a control surface with large 
displays to monitor nuclear reactors 

Industrial design 

Institute of Energy Technology 
(IFE) 

5 

A prototype of a control surface with large 
displays to monitor nuclear reactors 2 

Industrial design 

Institute of Energy Technology 
(IFE) 

6 

Working prototype of an environmentally 
friendly lamp 

Industrial design 

Noral Green Light 

7 

An evaluation of a self-driving robot vehicle 

Industrial design 

The University College 

8 

A Public displays design for giving concise 
information to employees, students, and 
visitors at the college. 

Interaction design 

The University College 

9 

A working prototype of a mobile applica¬ 
tion designed to give access to a library cat¬ 
alogue based on location. 

Interaction design 

Norwegian federal library, Deich- 
manske 


An example of an industrial design project made for the Norwegian University of Life Sciences is the proto¬ 
type of a veterinarian examination table on which dogs could be transported. The prototype focused on the 
transition from the use of finished profiles using steel to finished profiles using aluminium. Another indus¬ 
trial design example is the car designed for the Shell Eco Marathon race, a race challenging students around 
the world to design, build, test and drive ultra-energy-efficient vehicles (see Shell Global, 2017). As an inter¬ 
action design project, the students were challenged to create a new public display design for giving infor¬ 
mation via flat screens to students, employees and visitors at the university college. This project focused lay¬ 
out design in addition to extension of relevant support features such as video performance, interaction and 
impact for users. Figures 2-4 illustrate students working on their projects and examples of their prototypes. 


2 Group four and five, both worked on designing the same prototype. 
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Figure 2: Students developing different concepts design. 

In the capstone projects, the students attended discussions with their supervisor on an average of once a 
week. Little or no other teaching was provided outside of these critiques, a type of practice with long-stand¬ 
ing tradition in design fields (Chamorro-Koc, Scott, Coombs, 2015; Dannels & Martin, 2008). These critiques 
concerned various aspects of the students' work, changed in its style throughout the different phases of the 
project. Feedback could, for example, begin with getting the students started, ensuring that the project idea 
was related to their field of study, guiding the analysis, prototyping and testing, and giving feedback that en¬ 
sured the students adhered to what they said they would do. Later, an example of feedback could include 
supervising the final report and preparation for the EXPO. At the EXPO, the supervisors were particularly 
concerned that the students could discuss and evaluate their designs. The supervisors were concerned that 
the feedback they gave should be as objective as possible and somewhat independent of their own prefer¬ 
ences, or as one stated, ’Although I have my own taste or what I think is nice or working, you as a supervisor 
must try to give feedback relatively objectively. I as a teacher do not own the truth’. The feedback was given 
with the belief that the students themselves had main responsibility for their work and progress, and that 
the supervisors' role or desire was to create a learning environment that encouraged students to discuss rea¬ 
sons for the choices they had taken, as one supervisor explained, 

I ask the students why they have shaped the [product] as they have done, and this makes them take a 
standpoint on their decisions. It’s not enough to say that it’s pretty [...] The style of such supervisions is 
completely conscious, and I have learned this from my own supervisors. 

One of the industrial design supervisors was keen to convey that in this course, they ’teach design crafts, not 
design theory, which means discussions and feedback is on a practical level. Of course, the students' under¬ 
standing of conceptual theory is also part of the feedback’. Each main project received an individual grade, 
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A-E. For the most part, each student received the same project grade, as long as there was no differentia¬ 
tion of responsibilities and contributions that suggested that the grade should be given individually. 



Figure 3: Student working on the design to generate different concepts. 

The interviews occurred at the end of the course at the educational institutions in question, creating a sense 
of security and proximity to the topic of conversation. Having spent 14 weeks in these environments before 
conducting student interviews, I acquired sufficient knowledge of the practices used in the different pro¬ 
grammes, the professional language used by the supervisors and supervisee, the assignment given to the 
students and the feedback types used in the dialogues between the participants (see Karlsen & Karlsen, 
2012). This knowledge was used to develop an interview guide using a language the informants could relate 
to and understand. The individual semi-structured interviews were conducted to allow each individual stu¬ 
dent time and space to express his/her own viewpoints, reflections and opinions. To encourage this, I took a 
sensitive, open approach using confirmation and encouragement. However, in order to verify interpreta¬ 
tions and improve the interview response reliability, I also used a guiding, clarifying approach, requiring pre¬ 
cise, critical questions and interruptions (Kvale & Brinkmann, 2015). The interviews lasted approximately one 
hour. All were recorded and transcribed before being analysed. 
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Figure 4: Student developing a working prototype. 

In coding, data TAMS Analyzer software (Weinstein, 2006) was used. This software enables efficient coding, 
storage and reporting of coded information. To investigate the first research question of the types of feed¬ 
back students found useful, two initial categories were used: useful and useless. This dichotomy is an over¬ 
simplification that will miss much information about degrees of usefulness, but it gave an impression of 
which feedback patterns students found either useful or useless. The following excerpt illustrates a situation 
where feedback was perceived as useful: 

... we were about to start on the bachelor's degree project. We were going to make a design for a car, 
and we had one of those quick sessions with the supervisor where we sat and just had a total brain¬ 
storm. The supervisor brought up experience that he had gained from the previous projects about things 
we should do and things we should not do under any circumstances, and then we started to think. What 
I mean is, we immediately started to work and design on that basis, thinking "OK, we must avoid that" 
...and we arrived at the design a bit later, based on his idea that "Yes, I wish for you to manage some¬ 
thing creative and very asymmetrical". It was an idea that he had and that we considered and decided to 
proceed with. 

After classifying the feedback as either useful or useless, the PLUS model was used to further categorize and 
code the types of feedback. For example, we sat and just had a total brainstorm... was coded as 'brainstorm'. 
In addition, the feedback was coded as 'product', relating to the design of a car that they would create in the 
future, coded as 'feed-forward'. According to this student, the supervisor also wanted to promote their un¬ 
derstanding of how the product may look, coded as 'cognition'. This feedback was given the pattern 'brain¬ 
storm/product/cognition/feed-forward', which is in this case registered as 'useful'. In some cases, students' 
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descriptions were insufficiently detailed for the PLUS model to be used in full. Therefore, the analysis quality 
depended on good follow-up questions that could stimulate elaboration of the students' answers. Generally, 
short statements were easiest to code because they usually were coded with only one feedback pattern. 
Ambiguous statements were often most difficult to code, and in some cases a statement was given double 
codes. Druckman (2005, p. 263) states that coding is based on an interpretation process undertaken accord¬ 
ing to categories designed to identify elements as they occur in a text. Even though there are no right or 
wrong coding decisions, some are more plausible than others. 

In addition to plausibility, it is important to note that this study is based on a small sample size, 28 design 
students at one university college, and based on self-reported descriptions of oral feedback received during 
their bachelor projects. Caution must be applied, as the findings cannot be generalised to the supervision of 
students at other institutions, in other subjects or at other levels of study. Still, the results confirm some of 
what is already known from previous studies, which may indicate possible common challenges across feed¬ 
back situations, academic communities and traditions. 

A final methodological point must also be noted. Had the interviews been conducted immediately after feed¬ 
back was given and not at the end of the course, the results might have differed, as students' perceptions 
could change over time. Feedback they did not immediately consider useful might nevertheless be consid¬ 
ered useful later in the process. However, the opposite could also be the case. Students could forget over¬ 
time what feedback they received, particularly feedback that was not immediately deemed useful. In fact, 
students could actually be wrong about what was useful to them. What students think is useful is one thing; 
what is actually useful feedback is another. Jolly and Boud (2013) support this claim when stating, 'What stu¬ 
dents say they want might not be the same as what they need' (116). Having described the analytic proce¬ 
dures and methods used in this study, the next section is divided into two main parts, each section present¬ 
ing the results related to one of the research questions. 

Results 

The occurrence of useful/useless Feedback Patterns 

Fifty-three feedback patterns were identified using the PLUS model to code feedback comments. This means 
53 of the model's 192 possible patterns were found. Table 2 summarises the patterns most frequently de¬ 
scribed as useful or useless. To present the results systematically, the patterns are grouped by similarities 
when combining the model's three last aspects, focus, purpose and temporality (Table 2). Ten groups (col¬ 
umn 1) express 34 feedback patterns which represent those patterns that are mentioned by two or more 
students. The columns labelled 'useful' and 'useless' describe the number of students who found the feed¬ 
back either useful or useless. 

The most commonly occurring feedback patterns are in the groups 1-4 (column 1). In three of these groups, 
the most frequent patterns are coded as both useful and useless. All patterns in the four groups include cog¬ 
nition, meaning that the feedback is intended to strengthen students' intellectual abilities. The most com¬ 
mon feedback forms are corrections (patterns 2, 7 and 13), explanations (patterns 3 and 8), judgments (pat¬ 
terns 1 and 12) and suggestions (patterns 6 and 14). Feedback focusing on products found in groups 1 and 2 
is coded more frequently than feedback focusing on the students' learning process, groups 3-4. Feed-for¬ 
ward patterns are coded slightly more often than feed-back patterns. Of the 192 possible patterns, 139 
were not found in data. This is primarily because no statements were coded with 'psychomotor skills', which 
account for 64 patterns. In addition, only five of the 48 patterns that include self-regulation were identified. 

Interestingly, there is conclusive evidence that the feedback in this study is coded more useful than useless. 
Even though feedback patterns coded as 'product/cognition' found in groups 1-2 occur more frequently as 
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useful than feedback coded as 'process/cognition' found in groups 3-4, the number of patterns tagged as 
useless in groups 1-2 are also more frequent. The actual ratio between tagging the patterns as useful and 
the number of patterns identified is higher in 


TABLE 

2: 

OCCURRENCE OF USEFUL/l SELESS PATTERNS 



Group 

No 

. FORM 

FOCUS 

PURPOSE 

TEMPORALITY 

USEFUL 

USELESS 







No. of students 

No. of students 

1 

1 

Judgment 

Product 

Cognition 

Feed-back 

21 

16 


2 

Correction 

Product 

Cognition 

Feed-back 

16 

7 


3 

Explanation 

Product 

Cognition 

Feed-back 

13 

1 


4 

Emotives 

Product 

Cognition 

Feed-back 

9 

4 


5 

Questions 

Product 

Cognition 

Feed-back 

2 

4 

2 

6 

Suggestion 

Product 

Cognition 

Feed-forward 

20 

10 


7 

C orrection 

Product 

Cognition 

Feed-forward 

18 

12 


8 

Explanation 

Product 

Cognition 

Feed-forward 

17 

6 


9 

Questions 

Product 

Cognition 

Feed-forward 

6 

1 


10 

Emotives 

Product 

Cognition 

Feed-forward 

5 

2 


11 

B rainstorm 

Product 

Cognition 

Feed-forward 

4 

- 


12 

Judgment 

Product 

Cognition 

Feed-forward 

2 

1 

3 

13 

Correction 

Process 

Cognition 

Feed-forward 

19 

9 


14 

Suggestion 

Process 

Cognition 

Feed-forward 

16 

4 


15 

Explanation 

Process 

Cognition 

Feed-forward 

8 

3 


16 

Judgment 

Process 

Cognition 

Feed-forward 

4 

1 

4 

17 

Judgment 

Process 

Cognition 

Feed-back 

11 

1 


18 

Correction 

Process 

Cognition 

Feed-back 

2 

- 


19 

Emotives 

Process 

Cognition 

Feed-back 

2 

- 

5 

20 

Emotives 

Person 

Affection 

Feed-forward 

7 

- 


21 

Correction 

Person 

Affection 

Feed-forward 

5 

1 


22 

Judgment 

Person 

Affection 

Feed-forward 

3 

- 


23 

Explanation 

Person 

Affection 

Feed-forward 

2 

- 


24 

Questions 

Person 

Affection 

Feed-forward 

2 

- 

6 

25 

Emotives 

Person 

Affection 

Feed-back 

5 

3 


26 

Judgment 

Person 

Affection 

Feed-back 

3 

2 


27 

Correction 

Person 

Affection 

Feed-back 

2 

1 

7 

28 

Correction 

Person 

Cognition 

Feed-forward 

3 

- 


29 

Explanation 

Person 

Cognition 

Feed-forward 

2 

- 

8 

30 

Judgment 

Person 

Cognition 

Feed-back 

- 

2 


31 

Correction 

Person 

Cognition 

Feed-back 

- 

2 

9 

32 

Questions 

SRL 

Cognition 

Feed-forward 

2 

- 


33 

Correction 

SRL 

Cognition 

Feed-forward 

2 

- 

10 

34 

Emotives 

Process 

Affection 

Feed-forward 

2 

- 
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groups 3-4, with a ratio of 3:1, than in groupsl-2, with a ratio of 2:1. In other words, patterns in groups 3-4 
are coded three times more often as useful while the patterns in groups 1-2 are coded only twice as often as 
useful. In addition, a higher percentage of patterns coded as 'feed-forward' are tagged as useful in compari¬ 
son to patterns coded as 'feed-back'. Correspondingly, future-oriented patterns are coded as useless slightly 
more often than patterns coded as 'feed-back' which is a result of the total number of coded patterns being 
higher in patterns coded as 'feed-forward'. Judgments and corrections are the two feedback patterns most 
frequently coded as both useful and useless, while explanations and suggestions more often coded as only 
useful (see, e.g. patterns 3, 6 and 8). 

The above demonstrates the coding necessary to address the first research question of this study. The sec¬ 
ond research question can be addressed by framing in greater detail the conditions students deemed neces¬ 
sary for them to benefit from feedback. The following section is organized into three parts: the context in 
which the feedback pattern emerges, the students' confidence in the supervisor, and the students' own in¬ 
vestment in the feedback dialogue. 

Conditions for Feedback Patterns to be perceived as useful 

The feedback pattern found in group one, coded as 'judgment/product/cognition/feedback' is the pattern 
used to help identify the conditions presented in the following three parts. This feedback pattern was cho¬ 
sen because it was the most frequently coded pattern in all the interviews. 

The sequence in which the Feedback Pattern emerges 

The first condition necessary for students to consider feedback as useful depends on the sequence of which 
the pattern occurs and is independent of whether the comment is positive or negative ('this is good' or 'this 
is bad'). In other words, the context in which the pattern appears is important in determining how students 
perceive the feedback given by their supervisors. 'Totally disregarding the work done, with no constructive 
suggestions, is the worst type of feedback you can possibly get' is an example of a pattern coded as 'judg¬ 
ment/product/cognition/feedback' and considered useless. However, the following illustrates the same 
coded pattern as useful: 


STUDENT STATEMENT CODING 


1 

So it wasn't just like, "Yes, great! Go ahead!" 


2 

It was first, like, "Okay, why is that?" and "Why does it look like that?" 

[Question] 

3 

And then, I had to explain that, well, it is aerodynamic and it has a tear-drop 


4 

shape. And when we had to say, "Yes, we want to build it like this, and we 


5 

have bigger and simpler surfaces than before so that we can do it ourselves" 


6 

and then you can tell that they became more and more reassured that "wow, 

[Judgment] 

7 

they thought this through", and then they sort of followup by saying that 


8 

"yes, that, I think this is good", or, if something isn't good, [then they will 


9 

say that]; usually it's a bit of both, of course. 


10: And then they usually say that "you should think through and reconsider 

[Correction] 

11: that and maybe choose a different solution there" and things like that. 

[Suggestion] 

12: And I think that is a very good thing, that they give both criticism and 


13: encouragement, that is, in such supervision sessions. 



The above example illustrates a specific sequence of four patterns. The sequence of these patterns seems to 
affect the students' evaluation of the judgment given by the supervisor as relevant and meaningful. The 
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judgment described (lines 8-9) was not delivered in isolation but came after two questions (lines 2), fol¬ 
lowed by a correction (line 10) and a possible suggestion (line 11). In the first example where the judgment 
was considered useless, the judgement occurred in isolation, 'with no constructive suggestions'. The ab¬ 
sence of other patterns of feedback appears to be a reason the comment was considered as 'the worst type 
of feedback you can possibly get'. In a third example, one student explains why it is insufficient for a super¬ 
visor to just say 'fine', 'good' or 'not so good' by saying that 'you don't really learn anything, because you 
don't know what you did well or what you did badly so that you can do better next time'. The data clearly 
supports that the usefulness of the feedback pattern coded as 'judgment/product/cognition/feedback' is 
more dependent on the sequences of patterns than on the individual patterns themselves. Although the 
above examples use only the feedback pattern 'judgment/product/cognition/feedback', analysis indicate 
similar results for other feedback patterns as well. 

Confidence in the Supervisor 

In addition to the sequence in which the feedback pattern emerges, the usefulness of feedback was also re¬ 
lated to students' perception of the supervisors' personal characteristics. The following is an example of a 
student describing a supervisor as having high credentials: 

He is a highly merited designer, working in the field that we may also eventually work in... I think that I 
will remember him as a good supervisor after many years - that sounded almost like he's passed away - 
and as a person who had a lot to contribute and who, if you were in the middle of a situation as a de¬ 
signer, you might consider calling, simply for feedback. He would probably have appreciated that. So yes, 
I believe he has left his mark on most of us, including me. 

Based on data, two specific elements working together contribute to the perception of a supervisor. Is the 
supervisor sufficiently competent? This relates to his/her perceived knowledge, experience and qualifica¬ 
tions and his/her reputation and position. The other element is whether the supervisor takes his/her duties 
seriously. Do the students feel that the supervisor takes the time to familiarise himself/herself properly with 
the projects on which feedback is provided, and does the supervisor invest time, interest, care in the rela¬ 
tionship with the students. Below, describes an episode where a positive judgment (line 5) on a midway re¬ 
port was perceived as less good or useless feedback (line 7). These statements describe product-focused 
feedback delivered in the form of judgments, coded 'judgment/product/cognition/feed-back' (i.e. pattern 1, 
table 2). 
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LINE STUDENT STATEMENT _ 

1: We have talked a bit about how we have the impression that the supervisor may not 

2: have read what we submitted, that he maybe just skimmed through it. It is just a 

3: feeling, really, because we don't get any specific feedback [...]. It has just been sort 

4: of superficial. But we don't know. It may have been thoroughly read [...]. 

5: We got things like, "This was a good paragraph" and "There was an error here" [...]. 

6: It makes it more difficult to correct things, then, if it is sort of "okay", but maybe it 

7: isn't, really. So that makes it a bit more difficult afterwards perhaps [...]. 

8: It's not certain that it's his fault, in a way. It may be that we should have submitted 

9: things more often [...]. The supervisor said that we can just e-mail him stuff, but the 

10: times we have sent him things by e-mail, he has replied, like, a week later and things 

11: like that. 

12: [Interviewer: But you say that you only get "this is good". Do you feel that you can 

trust the feedback you receive? Do you trust what he says?] 

13: I have confidence in his knowledge and all that, so if I were convinced that he used 

14: all his experience and knowledge to actually go through everything we write—or 

15: maybe not everything , but anyway—then one might have more confidence in it... 

16: There is no doubt that he has a lot of knowledge. It's just that he may not be putting 

17: it to sufficient use regarding the work we submit [...]. 

18: He may be taking it a bit lightly. There has been a bit of groaning and huffing, a bit 

19: of body language, that suggests that it is an extra burden to be a supervisor, that it is 

20: something he would maybe prefer not to have to do. 


CODING 

[Not sufficiently 

thorough] 


[Judgment] 


[Own efforts] 
[Not sufficiently 
thorough/ 
interested] 


[Sufficiently 

competent] 

[Not sufficiently 
thorough] 

[Not sufficiently 
thorough'interested] 


In this example, the student considers the supervisor sufficiently competent, a person who can be trusted 
because of his knowledge and experience (line 13-14). However, because the student suspects the supervi¬ 
sor does not take his duties seriously enough (lines 1-3, 16-18) and because he demonstrates a clear lack of 
interest (lines 9-10,18-20); the judgment is not perceived as trustworthy. In a similar statement, another 
student dismisses much of what the supervisor says because she does not believe he has sufficient expertise 
(is 'sufficiently competent') on the topic. She always checks what he claims and that it has often proven to be 
wrong. She states the following: 'I guess that if you experience a person over a period of time, you get a cer¬ 
tain impression of what [the person] is like. That either builds trust or diminishes it. In this case, I have lost 
it'. 

Students' investment in the Feedback Process 

The final condition necessary to consider feedback as useful depends on the students' own investment in the 
feedback process. There are two particular elements that reoccur in the analysis: the student's effort in the 
process and his/her attitude. These two elements are illustrated by the following statement: 'I think that we 
are getting a lot of good feedback, that we are also aware of, really, and that you want to make use of...soon, 
but then, I think that we are just too lazy, or just have the wrong attitude'. The first element, effort, relates 
to the following factors, whether the students had prepared well for the supervision in advance (submitted a 
document to the supervisor, made an outline, created questions or determined who is to say what) and 
whether the members of the group actually attend the session, fit and rested. One student said: 

We have a certain responsibility ourselves, too, not just to have produced something but to have maybe 
thought about what we want from the supervision session and maybe also to have prepared more of an 
agenda for the meeting...It is we who are there to be helped, so then we have to find out what we need help 
with. 

Another student admitted they could probably be more active and persistent in dealing with the supervisor, 
if they felt they were not benefitting sufficiently from the feedback: 'Pretend that this is a school like Harvard 
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or something, where you pay a lot of money and expect to get... well, your money's worth, to put it simply. 
That is how you should think, really'. 

The second element, attitude towards the feedback process, is linked to student's expectations of feedback 
and to the extent of which the student trusts that the feedback won't be used against him/her. One student 
said, 'There have been days when my attitude has not been altogether positive, of course, and then I don't 
always want feedback. I don't know, but you have to try to keep a positive attitude, generally speaking'. 

Does the student dare to be open with the supervisor, not only about the strengths of the project but also its 
weaknesses? One third of those who were interviewed expressed concern that something stupid said during 
supervision would definitely affect their final grades. The following example displays how two students had 
very different approaches to supervision: 

I am happy to display [weaknesses], while my fellow student is quite the opposite, or that is my impres¬ 
sion at least. He wants to prepare for supervision, and then he wants to present the project as positive as 
possible. He does not want to draw attention to what is bad, and if our supervisor comments on some¬ 
thing that isn't particularly good, he defends it fiercely. We are very different because I think that what 
we say in supervision won't affect our grade. I'm the opposite. I am happy to draw attention to things 
that I'm uncertain about...I trust my supervisor that much. However, although I probably didn't trust the 
supervisor [we had in our first year], I still drew attention to things I was uncertain about then, too. 

The interview conducted with the 'fellow' student confirmed the differences. The 'fellow' student said, 'You 
don't want to show weakness. You want to reinforce the positive things to make the best possible impres¬ 
sion...eh...because then the supervisor will have a good impression of you. It will have a more or less direct 
effect on the grade, right?' Even though the student's attitude has no direct connection with a single feed¬ 
back pattern, it may be relevant to a particular session or to supervision in general. 

Based on the data, the two elements in the students' investment of the feedback process appear to have 
differing degrees of stability. The students' attitude appear more stable over time while the effort the stu¬ 
dents invest in the process may fluctuate throughout the process. 

Discussion 

Although McGrath et al. (2011, p. 1), suggest there exists a certain feedback type all students will find useful, 
a type therefore important for researchers to identify, the results of this research provide evidence that de¬ 
termining usefulness of feedback is more complex. This study demonstrates how one type of feedback can 
be perceived as both useful and less useful based on a variety of contextual elements, including the se¬ 
quence of the feedback pattern, students' confidence in their supervisors and the students' investment in 
the feedback process. The following section explores these results further. 

Useful feedback Patterns in the PLUS model 

The PLUS model can only describe what type of feedback is given expressed as patterns (Karlsen, 2015), but 
classifications can be used more normatively to identify and reflect upon feedback practices (Brown & 

Glover, 2006). I will use the identified feedback patterns in the PLUS model to discuss theories and research 
found on feedback in higher education, theories and research that have previously focused on the occur¬ 
rence of one aspect at time. Alternative to most reviewed studies on feedback in higher education, the PLUS 
model supports a richer description of feedback patterns as combinations of form, focus, purpose and tem¬ 
porality. However, due to lack of literature focusing on patterns of feedback, the following will discuss one 
aspect at a time. 
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The first aspect is the feedback form. In this study, feedback focusing on explanation was more frequently 
tagged as 'useful' and was rarely coded as 'useless'. Students appeared to appreciate explanations, which is 
supported by other previous research (e.g. Bailey, 2009; de Kleijn et al., 2013; McGrath et al., 2011). Accord¬ 
ing to Brown and Glover (2006), referring to Sadler (1989,1998), explanations create a connection between 
the feedback and the students' work, closing the gap between the current level of achievement and the de¬ 
sired level of achievement which is the core activity in formative assessment (p. 85). Similarity in research 
findings along with supporting theory suggest that feedback focusing on explanation is a valued component 
in effective feedback. Theory on feedback in higher education regards feedback in the form of judgement as 
useful in students' learning process (e.g. Boud & Molloy, 2013; Molloy et al., 2013; Sadler, 2010; Walker, 
2013). Students from this study supported these findings as they also relatively often assessed this form of 
feedback as useful. The low level of occurrences of the feedback forms ’brainstorming’ and ’interpretation’ 
in this study is another area that is supported by results of previous studies that also reported few accounts 
of these feedback forms. However, although these feedback forms were rarely identified, students consid¬ 
ered both the forms as helpful when they were first mentioned, a result that is confirmed by previous re¬ 
search (see e.g. Kumar & Stracke, 2007 discussion of 'interpretation'). 

The second aspect is focus on feedback. Similar to other studies, the students in this study frequently report 
receiving product-focused feedback assessed as useful (Hyland, 2001; Kumar & Stracke, 2007). Along with 
other research based on various levels of education, the present study seldom reports the value of feedback 
focusing on self-regulation as useful (de Kleijn et.al., 2013; Harris, Brown, & Hartnett, 2015). In the current 
study, only five (of 48 possible) patterns that include feedback focusing on self-regulation are coded, and 
these five patterns are mentioned only by 2 or less students each, which is a contrast to product-focused 
feedback that are coded in 22 of the 28 interviews. However, in return the five patterns that focus on self¬ 
regulation are all coded as useful. That only a few consider self-regulation as useful is surprising, as the im¬ 
portance of self-regulation is well documented in theory (Butler & Winne, 1995) and self-regulation is de¬ 
scribed as the core activity in sustainable feedback (e.g. Carless et.al., 2011; Price, Handley, O'Donovan, 

Rust, & Millar, 2013). Hattie and Timperley (2007) report feedback focusing on product (and person) is the 
least effective feedback, unless the feedback is directly attributable to self-regulation (p. 90). The results of 
this current study, therefore, contradict the accepted theories of effective feedback, theories that point to 
self-regulation as more effective than product focused feedback. Possible explanations for this rather contra¬ 
dictory result may be that students do not assess, characterise or conceive self-regulation in the received 
feedback, or perhaps lack the words to describe self-regulation. These findings indicate a need to better de¬ 
fine the concept of self-regulation in its use in theory and research. 

The third aspect is purpose. Students frequently considered feedback intended to strengthen their intellec¬ 
tual abilities (prompting reflection and cognition) as useful. Likewise, Lephalala and Pienaar (2008), in their 
impressive analysis of written feedback on essays, found feedback focusing on critical thinking and reasoning 
to be the most effective feedback, although this feedback was rarely given to the students. Previous studies 
report similar results (see, e.g. Brown & Glover, 2006; Ivanic, Clark, & Rimmershaw, 2000; Orsmond & Merry, 
2011). In theory on feedback, development of students' cognitive skills is found important to sustainable 
feedback but not more important than developing affective and psychometric abilities (Molloy, Borrell-Car- 
rio, & Epstein, 2013; Nicol & Macfarlane-Dick, 2006; Yang & Carless, 2013). Students in the current study 
similarly placed value on feedback encouraging affective skills, reporting a high incidence of two important 
efficient domains (cognition and affection). Lacking in the current study are reports on feedback coded as 
psychomotor. Physical (bodily-kinaesthetic) learning is central for professions like design (Schon, 1987), and 
as the sample comprised students designing and producing products like veterinary table for dogs, it would 
be expected to find such feedback in the results. Psychomotor feedback may not necessarily be expressed in 
words but through nonverbal communication (e.g. supervisor showing a specific grip to use a tool) which is 
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an aspect not included in PLUS model, which may also explain the results. This stresses the potential chal¬ 
lenges of capturing physicality or "embodiment" (cf. Merleau-Ponty, 1994) in research. The role language 
plays in bodily learning is an interesting topic for futures studies. 

The last aspect is temporality of feedback. Future-oriented feedback was often mentioned as useful in this 
current study, confirming what is already known about the effect of feed-forward (see, e.g. Brown & Glover, 
2006; Donovan, 2014; Hounsell, McCune, Hounsell, & Litjens, 2008; Price et al., 2010; Walker, 2013; Weaver, 
2006). Walker (2013), in her excellent investigation of students' responses to different types of written feed¬ 
back, explains that feedforward better enhances student-tutor dialogue, as feedforward provides improved 
’opportunities for students to adjust their future performance in response to comments received [and] 
therefore come closer to what Hounsell (2007) calls "sustainable feedback"’ (Walker, 2013, p. 108). As men¬ 
tioned earlier when discussing the first aspect, this last aspect supported by the combination of similar re¬ 
search and theory, confirms that feedforward is an important component in students' learning process. 

In this study, students reported benefiting from the categories explanation, product, cognition, and feed¬ 
forward. All four of these categories are supported by research as useful feedback. Three of the four catego¬ 
ries are supported by both research and theory as beneficial. Hypothetically, this could indicate the feedback 
pattern ’explanation/product/cognition/ feed-forward’ is useful feedback for this kind of supervision. None¬ 
theless, as we know from Table 2 (line 8), this pattern is not always perceived as useful, indicating that use¬ 
fulness depends also on the sequence of which the pattern occurs, varying confidence in the supervisor, and 
students' investments in the feedback process. The following further discusses the complexity of these con¬ 
textual conditions as reported in this study. 

The sequence of Patterns, trust in the Supervisor and Students' investment 

The first contextual condition indicates that the sequence in which a statement is made foresees the degree 
of utility more than the singular pattern itself, a claim also supported by Kjeldsen (2006): 

...it is not enough to praise and criticise. You must always give specific examples of what you praise or 
criticise, and say how it is good or bad [...] to exemplify how something could be done differently and say 
why that would be better (p. 166-167). 

Students in higher education seem to want precise, detailed feedback that allows them to improve their 
work (see, e.g. Carless, 2006; Ferguson, 2011; Lizzio & Wilson, 2008; McGrath et al., 2011). That said, few 
studies in higher education focus on the importance of the sequences for students' perceptions of benefit. 

An exception is the well-known effect of squeezing negative criticism between two positive comments (cf. 
Gardner, 2004; Hyland & Hyland, 2001; Kjeldsen, 2006), known as the feedback sandwich (Docheff, 1990; 
Molloy, 2010; Molloy et al., 2013) or the sugar-coated pill method (Dunworth & Sanchez, 2016). Mapping 
sequences can give a deeper understanding of how certain types of feedback may be more or less useful de¬ 
pending on what comes first and what comes after in one dialogue. More studies are needed to expand our 
understanding of how sequences of patterns relate to students' perceptions of the feedback given. 

The next contextual condition that is of equal importance for perceived utility depends on how much stu¬ 
dents trust the person delivering feedback. This confirms what is otherwise known about the importance of 
the supervisors' characters and how they present themselves (e.g. Chabaya, Chiome, & Chabaya, 2009; de 
Kleijn, Meijer, Pilot, & Brekelmans, 2014; Dysthe, Samara, & Westrheim, 2006; Kumar & Stracke, 2007; Lizzio 
& Wilson, 2008; Pitts, 2005; Poulos & Mahony, 2008; Price et al., 2010). Poulos and Mahony (2008) in their 
in-depth study of undergraduate students' perceptions of feedback, find 'the usefulness of feedback pro¬ 
vided and hence its credibility was related to the students' perceptions of the lecturers themselves (p. 145). 
Pitts' (2005) small-scale study discusses bachelor's students' willingness to take responsibility for improving 
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practice, and she links trust to assessments of how thorough a supervisor's work is—that the supervisor in¬ 
vests time and energy in familiarising him/herself with the students' work. Students see 'a scribbled com¬ 
ment' and 'hasty judgement' as an 'indication of lack of care and interest in their work, which lessens their 
trust in the tutor's professional judgement as well as potentially threatening their own self-confidence 
through carelessly negative comments (Pitts 2005, p. 221). Chabaya et al. (2009) use the expression gulf of 
mistrust (p. 219) to describe the mistrustful relationship between supervisor and supervisee identified in 
their mixed-method case study at Zimbabwe Open University. Dysthe et al. (2006) asked students what they 
believe is the most important success factors for feedback in supervision at the master's degree level at a 
Norwegian University. The students ranked trust, safety and sensitivity as most important. Dysthe et al. 
(2006) state 'This tells us that feedback has a very strong relational component that cannot be disregarded in 
any supervision context, particularly in groups' (p. 311). 

The concept of ethos used in rhetoric reflects a speaker's credibility. According to Kjeldsen and Torhell 
(2008), ethos 'is important to making people listen, and quite crucial when it comes to convincing them, be¬ 
cause we do not only consider what is said, but also by whom it is said' (p. 118). They argue that there are 
the three virtues used in ancient rhetoric—wisdom (competence), good character and goodwill—in relation 
to the recipients that enable us to 'appear convincing and credible even if certain dimensions of our ethos 
have been damaged' (p. 120). When a student in this study expressed 'I have confidence in his knowledge, so 
if I were convinced that he used all his experience and knowledge to actually go through everything we write 
[...] then one might have more confidence in it'. This statement is an example where the supervisor's 'good¬ 
will' has been damaged. Because of this, the student does not value the feedback, although the other two 
dimensions of the supervisor's ethos are considered strong. 

The third and final contextual condition is the students' investment in the feedback. The importance of stu¬ 
dent effort and attitude towards the feedback process are also endorsed by the research findings of Mirzaee 
and Hasrati (2014), who argue that feedback by itself is ineffective and that students 'need to reflect on, re¬ 
spond to, and act upon feedback for learning to occur' (p. 562). Higgins, Hartley, and Skelton (2002) found 
as many as 39% of students spend less than five minutes on received comments, implying a lost learning op¬ 
portunity, although other researchers (Donovan, 2014; Giles, Gilbert, & McNeill, 2013) suggest that most 
students do care about reading their feedback. This could be an indication that students' commitment to the 
role of a student varies. Rae and Cochrane (2008) separate students into two main categories based on their 
level of engagement. Some students make active use of feedback and are 'very keen to learn from it and de¬ 
velop academically. By contrast, other students seem to lack motivation and understanding, with a distinct 
lack of intent to learn' (p. 221). 

Rae and Cochrane (2008) not only categorise students in terms of engagement, they also identify both 
teachers and students confusing formative purposes of feedback with summative which in turn effects the 
students' attitude towards feedback. This confusion between formative and summative purposes of feed¬ 
back may be the basis of the mistrust mentioned earlier by the student reluctant to show weakness in his 
project during the feedback process. The notion 'that assessment and feedback is a transmission process 
centred on deriving and justifying a mark, rather than encouraging learning, seems to be predominate' (Rae 
& Cochrane, 2008, p. 221). Correspondingly, Li and De Luca's (2014) in their thorough review of 37 empirical 
studies notice 'tensions between formative and summative roles of assessment' (p. 390). Tease, Havnes and 
Lauvas (2005) state an important precondition for feedback to function optimally is that the system works as 
it should, officially as well as in reality (p. 65). As reported by Taasen et al. (2005, 65), it is naive for students 
to 'believe in the "formative talk" when it eventually emerges that there was a summative element involved 
after all'. In this perspective, reluctance in revealing weakness may be a beneficial approach to feedback in 
the long term. In the current study, as many as one third of the students may have been 'cue seekers', play¬ 
ing what Miller and Parlett (1974) name 'the examination game' (p. 59). Snyder's (1971) excellent work on 
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'hidden curriculum' also highlights these opaque, informal aspects of higher education assessment practices, 
drawing attention to potential 'hidden curriculum' containing messages that suggest strategies for academic 
survival and success that students pick up and pass to other students. These language games can include 
conflicts between teachers and students and contribute to negotiations based on fear and mistrust rather 
than dialogues based on trust (Snyder, 1971, p. 186). Snyder (1971) put forward the argument that, 'Where 
there is fear of exposing oneself...negotiation is likely to be the mode' (p. 186). Therefore, the hidden curric¬ 
ulum can easily work against the formal goals of professors and students (Snyder, 1971, p. 13) and render 
feedback situations less effective. Without a doubt, hidden practices should be articulated and addressed in 
the research aiming to understand and improve feedback practices. 

The above discussion has demonstrated how one type of feedback can be perceived as both useful and less 
useful. A variety of contextual elements, including the sequence of the feedback pattern, students' confi¬ 
dence in their supervisors and the students' investment in the feedback process must be met, according to 
these students, for them to benefit from feedback. 

Conclusion 

This study examines what feedback 28 bachelor's students doing their capstone projects report as useful. 

The study also explores factors on which they base their assessment of how useful the feedback is. Main 
findings are that students generally find supervisors' feedback more useful than useless and that the utility 
of this feedback varies qualitatively according to the sequence in which it occurs, supervisors' personal char¬ 
acteristics and perceived trustworthiness, and students' investment. That psychological aspects and stu¬ 
dents' efforts are important to their perceptions of feedback confirms previous findings, but the importance 
of the sequence in which a pattern occurs has not been extensively studied, and is an area for further re¬ 
search. In this study, students report less often feedback focusing on self-regulation, one of the most im¬ 
portant elements in sustainable feedback practice (Carless et al., 2011). This result is also confirmed in de 
Kleijn et al.'s (2013) results. The lack of students' focus on self-regulation could be due to reasons other than 
supervisors not providing such feedback. Supervisors perhaps could raise students' awareness of the useful¬ 
ness of self-regulation and endeavour to engage them in explicit dialogues about the usefulness of this feed¬ 
back type. This study does not provide a basis for drawing conclusions about the relative importance of the 
sequence of which the pattern occurs, the personal characteristics of supervisors or students' own invest¬ 
ments, and one can assume that several factors play a role and that the interaction between them is im¬ 
portant. For example, the importance of good cooperation in the student group as a factor that influences 
the perception of feedback is one example not considered in this article, although this factor was mentioned 
in the interviews. 

Based on this study, the key is to create good environments for feedback that take seriously situations in 
which students and teachers in higher education find themselves, situations that include time pressures and 
increased class sizes. These situations have contributed to the need to prioritize and understand what is 
beneficial supervision and feedback at this level. This article points out some of the conditions that may 
need to be met in order create these environments, however there is a need for more research-based 
knowledge that can contribute to further development of feedback practices in higher education, for 
example research addressing 'hidden practices' as discussed in this article. Another interesting path is to look 
into ways of providing feedback with the aim of psychomotor skills, an area especially relevant for practice- 
aesthetic subjects like design. A final example of a research area is the further exploration and development 
of supervisor and supervisee feedback literacy, enabling more 'meta reflection' on the feedback situation. 
This study demonstrates how the PLUS model can be used to identify and systematise factors that might 
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help students improve feedback utilisation in their learning process, a model that can be used in the further 
research needed in this area. 
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